Let ABC is an isosceles triangle inscribed in the ellipse

x² /a² + y² /b² = 1

such that point C lies on the end of the major axis and AC = BC.

Let coordinate of A and B are (a cosθ, b sinθ) and (a cosθ, -b sinθ) respectively.

Let A is the area of the inscribed triangle, then

      A = (AB * CD)/2

=> A = {2b sinθ * (a - a cosθ)}/2

=> A = ab*sinθ * (1 - cosθ) ............................1

Now differentiate with respect to θ, we get

      dA/dθ = ab[sinθ * sinθ + (1 - cosθ)*cosθ]

=> dA/dθ = ab(sin² θ + cosθ - cos² θ)

For maxima and minima,

      da/dθ = 0

=> ab(sin² θ + cosθ - cos² θ) = 0

=> cosθ - cos2θ = 0              (since cos² θ - sin² θ = cos2θ)

=> cos2θ = cosθ

=> 2θ = 2nΠ ± θ

=> θ = nΠ + θ/2 and θ = nΠ - θ/2  where n = 0, ±1, ±2,........

=> θ = 2Π/3 ∈ (0, Π)

Now d² A/dθ² = ab(2sinθ*cosθ - sinθ + 2cosθ*sinθ)

=> d² A/dθ² = ab(4sinθ*cosθ - sinθ)

=> d² A/dθ² = ab(2sin2θ - sinθ)                     (since sin2θ = 2sinθ*cosθ)

Now [d² A/dθ² ]_θ=2Π/3  < 0

Hense for θ = 2Π/3, A is maximum.

Now put θ = 2Π/3 in the equation1, we get the maximum area of the triangle

      A = ab*sin(2Π/3)*{1 - cos(2Π/3)}

=> A = ab*(√3/2)*(1 + 1/2)

=> A = ab*(√3/2)*(3/2)

=> A = (3√3/4)ab unit²

So area of an isosceles triangle inscribed in the ellipse is (3√3/4)ab unit²